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ABSTRACT 

An expanded program review process that includes 
qualitative, ethnographic, retrospective, and subjective data is 
described. At the Univer^'ity of Kansas, qualitative indicators were 
developed and methods were designed to implement them. Reviews were 
moved from a process of reacting to quantitative self-studies to a 
procedure that involved: intensive interviews, solicitation of alumni 
judgments on the merit and worth of their experiences (as opposed to 
collecting only limited-response surveys), and solicitation of 
reviews by external consultants :ho are disciplinary specialists. 
Results of the expanded reviews included development of qualitative 
data collection procedures, increased desire for narratives and case 
studies that contribute to greater appreciation of departmental and 
program constraints, understanding ot where flexibility might exist 
and where resources are strained, and the development of a "member 
check" process. This process verifies the accuracy of report/, checks 
for reliability, assesses the extent of assent with the findings and 
interpretations, and assures that all parties to the program review 
are fully informed of the report that is filed. Problems involved the 
training of faculty interviewers, commitment to the review, and 
coordinating with statewide reviews. (Author/SW) 
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ABSTRACT 

The need for comprehensive and strategic p*anning and reallocation 
in higher education has led many institutions to adopt internal 
program review as one Instrument to those ends. While program 
review is widely discssed on campus, neither the process nor the 
product is well understood by the faculty who are charged with 
carrying it out. • Furthermore, while faculty may be convinced that 
quality indicators extend well beyond quantitative data, they are 
at a loss to provide convincing qualitative data in support of their 
assessments. Consequently, program reviews often fall short of 
their intentions, to the disappointment and frustration of everyone 
concerned. 

This case study chronicles a program review process which extended 
the repertory of data collection methods to include qualitative, 
ethnographic, retrospective and "subjective" data. Efforts were 
made to identify qualitative indicators and to design methods and 
procedures for locating and incorporating them. Reviews were 
moved from a process of reacting to quantitative "self-studi'=*s" to a 
procedure - for utilizing intensive interviews, solicitation ot alumni 
judgments on the merit and worth of their experiences (as opposed 
.to collecting just limited-response surveys), and solicitation of 
reviews by external consultants who were disciplinary specialists. 

Results of the expanded reviews Included development of qualita- 
tive data collection procedures; greater appreciation of departmental 
and program constraints' understanding of where flexibility might 
exist and where resources were strained; and *he development of a 
"member check" process to verify accuracy of reports, to check 
for reliability, to assess the extent of assent with the findings a^ ' 
interpretations, and to assure* that all parties co the program 
review were fully informed of the report which was filed- 
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INDIGENOUS EFFORTS AT INDIVIDUALIZING PROGRAM 
REVIEW: A CASE STUDY 

Introtoduction : Bob Barak has just made the point (among 
others) that institutions tend to have more successful experiences 
with program reviews when those reviews are "...developed indigen- 
ously to meet the needs, traditions, and environment of a particular 
institution or system" (1986, p. , 22). That has been the experience 
at the University of Kansas, where program reviews are a fairly 
new phenomenon, and where faculty have not had long periods in 
which to grow accustomed to what the process means and to how 
it may best be conducted. 

As a result, the development of the process provides an 
intersting case study in the tailoring of a process to meet the 
needs of a particular campus at a particular point in time. 

As Barak has also noted in his history of the antecedents, in 
a very short period of time the purpose of the original program 
review process has shifted from one of "program improvement" 
toward that of "program decision making, including termination" 
(1986, p. 15). In part, this is a retlection not of campus interest, 
but rather of fiscal and programmatic mandate, as resources have 
grown scarcer, and as the need for comprehensive and strategic 
planning and reallocation in higher education have grown more 
pressing . 

History of Prog ra m Review at the U niver sity 
A frmal process for conducting program review at the 
University of Kansas was instituted in the Autumn of 1980, after 
much discussion within and among faculty governance groups, the 
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Faculty Senate and the Senate Executive Courcil, and the Faculty 
Council . 

Historical Changes in the Proces s 
Early Efforts » The first several rounds of program reviews at the 
University were rooted in three elements: a departmental "self- 
study", the ETS faculty and student survey, and graduate faculty 
review. The departmental self-study was similar to, although not 
as extensive as, a typical accreditation self-s .dy. It was rooted 
in indicators of criteria related to cost, productivity (including 
credit-hour production at the graduate and undergr. iuate levels), 
need and demand for the program, and reputation ratings, which 
were generally collected from such sources as the C hronicle of 
Higher Education's annual rankings. The latter were especially 
utilized if they demonstrated that the department of unit ranked 
among the top 100 in the country. Problems with "prestige" 
rankings of graduate programs particularly were not debated or 
discussed (Conrad and Blackburn, 1984?). 

The ETS faculty and student survey was routi-.iely distributed 
by the department to all faculty and to some sample of students, 
all of whom were instructed- to return their completed forms to the 
Graduate School, which in turn forwarded them for process to 
ETS. Both the department and the graduate program review 
comrf)5ttee had access to the results of this survey when it was 
returned to the institution after processing. Early on in the 
process, the surveys were used extensively as "r\/idence" of 
performance or non -performance on the part of the department in 
various arenas (e.g., graduate student advising). Later on, after 



1982, the surveys continued to be used, but came to be considered 
one of many forms of data available. As a result, the surveys 
receded in importance for many departmental reviews. 

The third element, graduate faculty review, was directed 
toward maintaining and improving the quality of graduate instruc- 
tion by reviewing the research productivity records of individual 
faculty members. Two outcomes were possible from this review of 
current vitae: the faculty member could be 'Ve-certified" for 
membership on the Graduate Faculty (and therefore, continue to 
chair and advise doctoral dissertations), or the faculty member 
could be recommended for removal from the Graduate Faculty 
roster (for a period of five years). The philosophy behind such 
review was tht faculty who themselves were not engaged in active 
research programs were probably unable to direct the research 
efforts of graduate students. 

Of the institutional elements of program review, this was 
probably the one most honored more in the breach than in the 
observance. In the early years and even now, faculty from other 
departments found it difficult to recommend that other faculty 
should have their dissertation chairing privileges revoked for some 
period of time. As a result, the Graduate School (which directs 
program reviws) has often had to request the information and 
recommendations as additional items once the review report has 
been turned in. And equally often, the recommendations have 
been aimed at wholesafe re-certification, despite the apparent 
Inactivity of some faculty members in research. The judgment is 
often rendered that the committee has no way of judging what 
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constitutes productivity in a discipline outside that of the committee 
members, and that time-lines for publication are sufficiently disparate 
that it is difficult to tell whether or not a faculty member in the 
reviewed department is active or not. 

In part, this problem is solved by the second review cycle, 
which is conducted at 10-year intervals, and directed by outside 
consultants (although as yet, no 10-year cycles have come around). 
It is believed that disciplinary specialists outside the institution 
may be firmer in their judgments regarding productivity, and can 
render decisions regarding how often and in^what outlets faculty 
members should be presenting their work.^ 

Efforts after 1982 . The author was asked to take the place 
of a program review chairman whose sabbatical and other duties 
prevented him from completing the review. After extensive consul- 
tations with Graduate School staff members, review of other program 
review reports completed within the past year, and review of the 
literature on program review in other institutions and states, it 
was collaboratively determined that the evaluation and program 
review effort should be expanded. The proposed expansion was 
predicated on my own experience with program evaluation, a partic- 
ular philosophy of, what comprised good evaluation data, and the 
Graduate School need for more comprehensive treatment of program 
resources, constraints and flexibility. 



more complete explanation of the 5-year and 10-year review cycles is 
contained in the " 1980, ^University o^ Kansas. 
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My own experience with program evaluation suggesteu that 
evaluation and program review efforts were often incomplete and 
decisionmakers felt frustrated with techical and quantitative data 
alone. Decisionmakers intuited--quite righhtly--that numerical data 
failed to paint a complete picture, and that other data might be 
equally commanding if they were but available. The problem was a 
two-pronged one: first, the confusion of "criteria'' with "indicators" 
for that criterion (Barak, 1986, p. 19), and second, appropriate 
methods and procedures for gathering indicator data. 

The particular philosophy of program evaluation which guided 
the efort derived from naturalistic inquiry (Guba and Lincoln, 
1981; Lincoln and Guba, 1985; Guba and Lincoln, 1982), rather 
than from conventional inquiry, or traditional s .ntifx and technical 
formats for program evaluations. Several axioms of naturalistic 
inquiry were most useful for program reviews. These included: 
1) the assumption that reality is not singular, but is rather an 
individually - and group-centered construction, and therefore 
multiple, intangible, holistic, and indivisible into separjte compon- 
ents for the purposes of study; 2) that linear cause-effect relation- 
ships are impossible to establish in the social/behavioral world, 
and that therefore, inquiry and evaluation ought rather to focus 
on plausible inferences, webs of relationships, pattern theories, 
and miilual, simultaneous shaping ralhern than sirairjhl -line, 
"If-then" statements; and, finally 3) that higher education is a 
value-laden enterprise, so that the demand for objectivity is a 
useless call, and that more valuable for decision-making would be 
the exploration of multiple value and belief systems by disinterested 
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third parties. This final assumption, besides embedding the 
program review in value considerations relevant to the institution, 
had the effect of grounding the review in loca.' history, local 
tradition, and statewide needs assessment. 

The implications of a move to new-paradigm (naturalistic) 
philosophical assumptions were that such assumptions called for, 
and could make unique use of qualitative methods and strategies 
in me program review design. A small proposal was written which 
outlined the undergirding philosophy for this particular program 
review, and which proposed procedures for accomplishing the 
review utilizing both quantitative and qualitative data. 

The graduate school met thhe proposal with interest and a 
measure of reserved enthusiasm. Their own neeas for additional 
•data although not certain what kind and for a broader treat- 
ment of program and departmental quality, demand/need and 
cenlrality, as well as a fuller understanding of resources and 
constraints, were not being met. Much of the data which had 
comprised the reports between 1980 and 1982 were those which 
were recoverable (inslightly different form) from the Office of 
Institutional Research, from accreditation reports, and from other 
sources on campus. The Graduate Scnool also felt that wider 
participation in program review was necessary. Typically, reports 
had focussed on the data provided by only a few faculty members, 
or in smaller departments, by the chairperson alone. The review 
proposed in this instance^ incorporated four new elements not 

^The program review undertaken did not in fact involve 'a department, 
but rather a "program area": an area-studies program to whic!i faculty 
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utilized previouslyr .nlensive and unstructured interviews of aM 
faculty members affiliated with the program; solicitation of alumni 
judgments regarding the utility and vilue of their experiences on 
campus; solicitation of reviews by external disciplinary specialists; 
and the development of a comprehensive "member check" process to 
ascertain accuracy of final data and degree of assent to the review 
findings and interpretations. 

D escription of the process . Typically, a chairperson is 
designated, and faculty members (usually 3-5 in number) are 
requested to join the effort. These faculty members are chosen 
from outside the departmental ranks, and typically represent, 
insofar as possible, both Arts and Sciences nd professional schools 
perspectives. In this case, four persons and a chair were desig- 
nated. In addition, the department undergoing program review is 
asked to designate a "faculty liaison"--a person who knows the 
department well, and who can obtain and present data which the 
committee might request. In this case, both a senior faculty 
member and the program chairman participated as liaisons to the 
committee. Convening of the committee and its liaisons is the 
responsibility of the pft:;gram review chairperson. 

Members of the committee and liaisons were given copies of 
earlier program reviews, asked to study them, and were oriented 



'were "lent" from regular departments ro to which faculty affiliated 
themselves because of their own scholarly and teaching interests. 
The program area was Soviet and East European Studies, and 
faculty were affiliated from such disparate areas as political science 
and theatre, geology and anthropology, and Germanic studies and 
religion. Twenty-six faculty in all were involved, under the 
direction of a program chairman who coordinated Federal grants, 
military projects with local and state militar/ bases, travel, library 
acquisitions and other department-liKe functions. 
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to differences in the process which they would undertake. Thp 
four additional processes intensive interviewing, solicitation of 
alumni judgments, solicitation of external reviews and the final 

Mdity check were explained to them, and they agreed to 
experiment with the process as devised. Tne liaisons were given 
responsibility for providing certain forms of data to each of the 
committe members for review (including federal proposals, graduate 
program course listings and descriptions, lists of personnel assigned 
to the program area and percentages of responsibility, and the 
like), and committee members took responsibility for interviewing 
eight or more program area faculty during the course of the study. 

The next step was to create intt.view protocols which would 
at the same time leave ample room for the constructions of faculty 
in the program and yet cover major areas of interest in a program 
review. The committee and liaisons worked several sessions in a 
nomination process until an outline was develped which covered all 
aspects of program functioning, resources, administration and 
future plans which could be conceived. When the protocol was felt 
to be fairly complete but still open-ended enough, committee members 
undertook to contact their interviewees, ^et up interview times, 
and complete the interviews. 

During this same time period, a sample letter to alumni was 
prepared and distributed for revision and refinement. After 
several revisions, the letter was sent to alumni who had majored or 
minored in the program area during the past ten years'. They 
were asked to describe their current positions, to describe how 
and in what ways their program studies as a value aaded to their 

12 



10 

curr'^nt positions, and to comment on the unique contributions of 
any faculty member who proved especially helpful either during 
th^i- studies, in pb^ement, or later in their careers. Of some 
twenty letters sent, seven responses were Forthcoming, While this 
was not a high return rate, several were returned as undeliverable 
because of address changes, and those wh) h were returned proved 
thoughtful and balanced in their assessments, particularly in the 
area of what the program contributed to the individucil's professional 
preparation. Return ra, might also have been low due to the 
respondents' being asked to complete the ETS survey furm for 
graduate students. 

Also during this period, ailcUe^ was drafted to solicit responses 
from other disciplinary specialists. The letter was not sent, 
however, in a .Ucipation of the state Board of Regents' review, 
which was take placr the following year. Since the same information 
would be solicited at that time, it was determined that while this 
was a useful form of data to collect, particularly at the 5-year 
review cycle (when outside consultants would not be brought in), 
in this instance, it was not collected. 

When all faculty members, many graduate students, and 
program research assistants and secretaries had been interviewed 
r^Atensi yely, the committee met without the faculty liaisons to 
pnqnqn In dcbrioring--lhe shnring bolwp<»n . commillc'o mcMiilxM'*"* o( 
themes and data to support interpretations from the analysis of 
individual interviews. These debriefing sessions (three, in all) 
were taped, and committee members provided to the chairperson 
their own interview notes. After reviewing the tapes and interview 
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notes, the chairperson convened the committee again to determine 
responsibility for writing up sections of the final report. Each 
committee mrember, including the^chaii^man, took one cr more thematic 
area and constructed a draft from his or her own notes and the 
notes of other committee members. 

Final drafts were the responsibility of the committee chairperson, 
who undertook to smooth out stylistic differences between comnlittee 
memoers* writing, to augment each section with additional quotations, 
numerical data and other evidence, and to provide a smooth flow to 
the narrative. lnterpreta:ions and final recommendations were 
created from the individual sections or themes, and the first part 
of the member-cherk process begap. 

In the first stage of that process, committee members read 
and re-re?id drafts to be certain that editing did not change their 
original intentions, and that they were satisfied with the interpre- 
tations which thev and others had made. They also read carefully 
CO determine whether the evidence advanced was the best which 
could be mounted for a given finding or interpretation, or whether 
their own notes provided either additional or more compelling 
evidence which might be substituted. Wheri the committee was 
satisfied that the document was a fair and balanced as they could 
make it, it passed for review to the faculty liaisons and to the 
program area secretary (who had immediate access to budgetary 
figures and could check dollar amounts, travel funds and personnel 
appointments). 

The program area chairpersrn m particular read the draft 
document with a careful eye, ?nd sugge5tions for editing and 
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correcting v/hich he advised were acceptea by the review committee 
without any difficulty. . Finally, the program area faculty were 
notiffed by memo from the chairperson of the committee thaat a 
copy ot the draft reviewed by the committee and by their 
appointed 'liaisons-- would be circulating to them soon, for their 
revirw and comments. Each person interviewed (two faculty members 
were not interviewed, as they were on sabbatical during the Fall 
semester) was given the right to withdraw any quctatibn (non- 
attributed) which he or she made and wh'ch was included in the 
document. This round of member checks drew several memoranda 
of appreciation for being able to review the draft, and compliments 
on its balance and thoroughness. 

The final draft was prepared and forwarded to the Graduate 
School in the Spring of 1983. Included in the final program review 
report was a .section on "Methodology", explaining what procedures 
were utilized to gather what kinds of data for which indicators and 
which criteria those indicators were thought to "ddress. The 
dec'sion of Graduate School staff ws to use tHe final report as a 
model for future program review efforts, and the report is now 
sent when new chairpersons ask for examples of such reports. 

Outcomes of the Expanded Program Review Process 
There were four major outcomes of the expanded program 
review process, and each of those contributed to the buttressing 
of prdgram review as a useful and productive activity for both 
faculty and the Graduate School. Those four outcomes were the 
development of some qualitative data collection procedures; the 
creation of a greater desire for narratives which contributed to 
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larger appreciation of program constraints; the development of an 
understanding of where programs and departments might have 
flexibility, or where programmatic flexioility mighi exist; and in 
the Development of a member check process. 

Qualitative data collection procedures . The data collection procedures 
used here were not particularly sophisticated, but they did represent 
the first attempt to incorporate su/i data into the program review 
process, and they represented too the first time interviews of an 
entire program area's or department's faculty were conducted. 
The solicitation of alumni "value-added" comments on their programs 
was a major extension of the usual data-gathering procedures for 
alumni, and while the response rate was low, those responses we 
received were articulate and thoughtful commentaries. . Some faculty 
expressed the idea that qualitative data might be overly "subjective', 
but the committee made the argument in return that strictly quanti- 
tative data were also subjective, and that the choice of which 
numerical data to use, and the format for displ':jy were value 
choices. The main difference between the subjectivity bf quafita- 
tive and ouantitati\e data was the participation of all stakeholders 
in determining the appropriate balance for all data . Stakeholders 
as well as rcviirw committee members had a chance to contribute to 
the array of evidence offered in support of any interpretation or 
recommendation. 

Appreciation of program constraints . Because a narrative form 
and case study style illow for the presentation of conflict and 
tension, the dynamics of operating a program area where nearly all 
time is contributed, and maintaining continuity of program offerings 
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for graduate study could be demonstrated. Staff of the Graduate 
School commented that they were able to see and understand 
problems more cleariy with his reporting format than with other 
more technical and numerical forms. Trade-offs in running such a 
program (not to mention the inevitable tension in generating outside 
funding for support services and library acquisitions) were amenable 
to the subtle counter-argument which narrative style affords. 
Constraints displayed in this manner were simply better appreciated 
by those who would ultimately make decisions about program improve- 
ment or continuation. » 

^ The search for flexibility . A second advantage to the case study 
approach utilizing qualitative data is that recommendations do not 
come as a surprise to the reader. When constraints on the program 
^re disc-usred as dynamic tensions, the recommendation for more 
resources, or for a reallocation of resouurces to areas of greater 
need simply "drop out** as part of the larger understandings. 
When resources are flexible but can only be rearrayed' with admin- 
istrative approval, the rationale for doing ro ib many times clearer 
;n case study format. 

A "member-check" process . One of the most appreciated of the 
innovations in the process appeared to be the member check pro- 
cedure, whereby all interviewed facuky and staff were furnished a 
draft of the final report, and Iheir rf^acMofis were solicilod. Some 
faculty took the opportunity to correct figures or facts, although 
not many were incorrect (or mis-typed); some took the chance to 
press for additional recommendations (primarily 'jr additional 
, resources); and many commented that they appreciated having the 
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chance to read the document which concerned them and that they 
thought it fair, balanced, reasonable and accurate. Typically, 
faculty do not read the final draft until it has been sent to the 
Graduate School, and equally typically, they have no say in its 
final format or findings. 

On the principle that people "own" the data which they furnish 
about themselves, each faculty member also had the right to withdraw 
specific comments which he or she made (which were used as 
illustrative of findings and interpretations). No faculty member 
withdrew any comment, but many complimented the review committee 
on its fairness in presenting such qualitiative data. 

While the member check process stretched the time for review 
out for another 4-6 weeks^ it seemed well worth the trouble to 
allow faculty members to review and reflect on wh ,t they valued 
their affiliation with this program area. 

Problems with the Process 

As Barak has also pointed out, the program review process 
itself may prove, over time and especially with shifting purposes, 
to be inadequate, and ought to be periodically evaluated for "fine 
tuning" (1986, p. 24). There were and still are four problems 
with the program review process at the University of Kansas. 
Each of those can alter, extend, or utterly sabotage the process. 
Since they are not simply institutional problems, but may extend to 
other institutions and other settings, each deserves a mention. 

The f!rst problem lies with the training of faculty to be 
interviewers. In the case presented here, all of the committee 
members had done interviewing (as a formal itiquiry process) 
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previously; most of them in their own research work. But in 
other circumstances and with other faculty the skills needed for 
careful but unstructured interviewing and the maintenance of field 
notes might be foreign or threatening. Fortunately, most faculty 
have served in governance work (such as search and screen 
efforts) where interviewing is used. But the committee chairperson 
might have to make the link more to go •"nance than to research 
and inquiry more generally, since many faculty would not have 
used interviewing necessarily in their own inquiry arena. 

Second, there is occasionally (or perhaps cTten) a commitment 
prpblem. The necessity for the chaihperson to take seriously the 
prograhi review assignment, and to be persistent in convening the 
committee and liaisons is underrated. The committee will not 
convene itself. Unless the committee is headed by a person who 
takes the program review process seriously, and who is willing to 
devote personal or secretarial time to meeting the scheduling 
demands of busy faculty, the job will not be done, or it will be 
done indifferently. The program review comissioning body (in 
this case, the Dean of the Graduate School and his staff) must 
exercise great care in the choice of the chairperson, or some 
reviews wUI be done tardily if at all. 

A third problem is also a quaint one. It has to do with 

departments who unilaterally decide they are "too busy" to enter 

into the program review process, or that they have too many 

problems at the moment to undergo review, that a concurrent 

accreditation is taking too much of their time, or the like. Several 
t 

departments in this same university have in effect refused to 
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cooperate in the review process. In the face of their collective 
stubbornness, the process has broken down. What can be done 
about thatt problem is unclear. When an academic department 
Vefuses to admit thei review committee and announces its intention 
to subvert the process, the response tends to be one of bafflement 
and confusion. The resolution of this recalcitrance awaits a more 
ingenious mind. 

The fourth problem is that of coordinating the program review 
process on campus with the statewide coordinating agencv's review 
process. In some instances, including the case presented here, 
the/campus program review process was scheduled either to precede 
or roughly coincide with the statewide review. Fortunately, the 
statewide review took this particular case study as illuminating, 
but the decreased reliance on technical a^id numerical data meant 
that the program area had to supply a second self-study, essentially. 
,ln the future, it mJght be wise to have the program review final 
draft include more technical data, which would then not have to be 
re-arrayed and represented to Jthe statewide coordinating agency. 
Providing the same data again and again in different formats leads 
to the feeling of being "studied to death'*. It seems a shame to 
review programs which clearly ought to be continued and supported 
until they are drowned in the data-collection and review effort. 
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